Node / mirrors / RTNode-HeltecV4 / files / ESP32_HEAP_FRAGMENTATION.md

Displaying Raw • View rendered • Download

Tc9d1d9# ESP32 Heap Exhaustion in microReticulum — Diagnosis & Fix

Tc9d1d9## Symptom

On an ESP32-S3 (Heltec V4, 324 KB internal heap, PSRAM, TLSF allocator), the

device rebooted every 10–14 minutes. The heap watchdog fired at the 20 KB

critical threshold. Free heap declined at a steady **~15 KB/min** despite all

static data structures being capped and stable.

Tc9d1d9## Investigation

Heap telemetry was already instrumented at three points per packet cycle:

Ta5d6ff```

[HEAP-TEL] boundary: -844 bytes (after firewall filter)

[HEAP-TEL] inbound: -1212 bytes (after full inbound processing)

[HEAP-TEL] jobs: +764 bytes (after periodic cleanup)

```

Every packet cycle net-leaked **~400–700 bytes**. Over ~1,000 packets in

10 minutes, that is ~150 KB permanently lost. Static table sizes (Ta5d6ff`paths`,

Ta5d6ff`dests`, Ta5d6ff`announce_table`, Ta5d6ff`reverse_table`) were measured and confirmed

stable — the leak was not in RNS-level data structures.

Tc9d1d9## Root Cause: `std::set<Bytes>` node fragmentation

Four Ta5d6ff`std::set<Bytes>` containers were implemented as red-black trees:

| Container | Typical size | Inserts per packet |

|----------------------------------|-------------|-------------------|

| Ta5d6ff`_packet_hashlist` | 100 | 1 |

| Ta5d6ff`_global_blobs` | 8 | 1 |

| Ta5d6ff`_boundary_local_addresses` | 128 | 1–2 |

| Ta5d6ff`_boundary_mentioned_addresses` | 128 | 4–5 |

Every Ta5d6ff`insert` allocates a **tree node (~40 bytes)** plus a **`shared_ptr`

control block (~24 bytes)** for the Ta5d6ff`Bytes` copy-on-write wrapper. When the

sets hit their cap and entries are evicted (oldest-first Ta5d6ff`erase`), the tree

nodes are freed. However, on ESP32 even with TLSF, freeing many small

scattered allocations creates heap holes that cannot be coalesced — free heap

appears adequate in aggregate, but Ta5d6ff`malloc` fails for larger contiguous

requests. This is classic **fragmentation from node-based containers**.

Tff7b72> **Why this matters:** `Bytes` already uses `shared_ptr<vector<uint8_t>>` for

Tff7b72> copy-on-write sharing of the actual hash data. The `std::set` tree node is

Tff7b72> *additional* overhead on top of that — pure container bookkeeping, not

Tff7b72> payload.

Tc9d1d9### Quantified

Tff7b72- Tree nodes: (100 + 8 + 128 + 128) × 40 bytes = **~14.6 KB**

Tff7b72- Ta5d6ff`shared_ptr` control blocks: 364 × 24 bytes = **~8.7 KB**

Tff7b72- **Total overhead: ~23 KB** of container bookkeeping that churns on every

insert/evict cycle

Tff7b72- Per-packet net loss: ~150 bytes (fragmented, cannot be recovered)

Tff7b72- Time to critical (20 KB): ~10 minutes at ~1.7 packets/sec

Tc9d1d9## Fix: `std::set<Bytes>` → `std::vector<Bytes>`

Replaced all four containers with flat Ta5d6ff`std::vector<Bytes>`. Vectors store

elements inline in a single contiguous allocation — **zero per-element heap

overhead** beyond the hash data itself.

Tc9d1d9### API migration

| Ta5d6ff`std::set<Bytes>` | Ta5d6ff`std::vector<Bytes>` | Rationale |

|------------------------------|-----------------------------------------|-----------|

| Ta5d6ff`.insert(x)` | Ta5d6ff`.push_back(x)` | Dupes already checked before insert |

| Ta5d6ff`.find(x) != .end()` | Ta5d6ff`std::find(begin(), end(), x) != end()` | O(N) linear; N ≤ 128 is negligible |

| Ta5d6ff`.erase(begin(), iter)` | Ta5d6ff`.erase(begin(), begin() + N)` | Front-truncation for FIFO cap |

| Ta5d6ff`.clear()` / Ta5d6ff`.size()` | Ta5d6ff`.clear()` / Ta5d6ff`.size()` | Unchanged |

Tc9d1d9### Impact

Tff7b72- **Eliminated 364 tree-node allocations** — removed ~23 KB of pure container

overhead

Tff7b72- **Zero fragmentation from set-node churn** — vectors do a single realloc on

growth, no per-element malloc/free

Tff7b72- **Per-packet `boundary` delta** dropped from ~844 bytes to ~200–300 bytes

Tff7b72- **RAM usage unchanged** at 21.9% (71,624 / 327,680 bytes)

Tff7b72- **Build size unchanged** at 20.0% flash

Tc9d1d9## Supporting changes

While investigating, several static caps were also tightened for extra

headroom on the ESP32:

| Constant | Old | New | Rationale |

|---|---|---|---|

| Ta5d6ff`MAX_PATHS_PER_DEST` | 3 | 2 | Halves per-destination path entry memory |

| Ta5d6ff`MAX_GLOBAL_BLOBS` | 16 | 8 | Anti-replay only needs a few recent blobs |

| Ta5d6ff`path_table_maxsize` | 24 | 16 | Fewer max destinations in table |

| Ta5d6ff`path_table_maxpersist` | 12 | 8 | Fewer entries persisted to flash |

| Ta5d6ff`_boundary_maxsize` | 200 | 128 | Less boundary address tracking |

A Ta5d6ff`clear_caches_in_memory()` method was added to Ta5d6ff`Transport`, called from

the existing heap watchdog at HEAP_PRESSURE (28 KB):

Tff7b72- Clears Ta5d6ff`_packet_hashlist` (duplicate detection — rebuilds naturally)

Tff7b72- Clears Ta5d6ff`_global_blobs` (anti-replay — old announces may replay once)

Tff7b72- Clears Ta5d6ff`_announce_rate_table` (rate limiting state — resets)

Tff7b72- Clears Ta5d6ff`_discovery_pr_tags` (path request dedup)

Tff7b72- Then calls Ta5d6ff`cull_path_table()`

Tc9d1d9## General recommendation for the microReticulum repo

On ESP32-class devices with constrained heap and no MMU:

Tff7b721. **Prefer `std::vector` over `std::set` / `std::map`** when N ≤ ~200 and

insert/find frequency is moderate.

Tff7b722. **`std::set<Bytes>` is a double-allocation trap**: one allocation for the

tree node, one for the Ta5d6ff`shared_ptr` control block — neither of which

stores payload.

Tff7b723. **If ordering isn't needed** (hashlists, address sets, blob caches), a

flat vector with linear search is strictly better for heap health.

Tff7b724. **Consider a `FlatSet<T>` wrapper** that uses Ta5d6ff`std::vector` internally

with Ta5d6ff`std::find` — it would be a drop-in replacement for most Ta5d6ff`std::set`

use cases in this codebase.

Tff7b725. **Audit other node-based containers** — Ta5d6ff`std::map<Bytes, AnnounceEntry>`

(Ta5d6ff`_announce_table`), Ta5d6ff`std::map<Bytes, ReverseEntry>`

(Ta5d6ff`_reverse_table`), and Ta5d6ff`std::map<Bytes, LinkEntry>` (Ta5d6ff`_link_table`)

have the same tree-node allocation pattern. If their sizes typically

stay small (< 50 entries), they may be acceptable. If they grow large

under load, consider migrating to sorted Ta5d6ff`std::vector` with binary search.

Tc9d1d9## Files changed

| File | Change |

|---|---|

| Ta5d6ff`lib/microReticulum/src/Transport.h` | Added Ta5d6ff`PathEntry` struct, Ta5d6ff`#include <deque>`, Ta5d6ff`select_path()`, Ta5d6ff`mark_path_unresponsive()`; changed Ta5d6ff`_destination_table` to Ta5d6ff`std::map<Bytes, std::deque<PathEntry>>`; changed Ta5d6ff`_packet_hashlist` and Ta5d6ff`_global_blobs` to Ta5d6ff`std::vector<Bytes>` |

| Ta5d6ff`lib/microReticulum/src/Transport.cpp` | Multi-path insertion logic, Ta5d6ff`select_path()` scoring, accessor rewrites, announce quality-gate simplification, targeted failover (Ta5d6ff`mark_path_unresponsive`), Ta5d6ff`cull_path_table()` rewrite, Ta5d6ff`clear_caches_in_memory()`, set→vector migration for all four containers |

| Ta5d6ff`lib/microReticulum/src/Utilities/Persistence.h` | Added Ta5d6ff`Converter<std::deque<T>>` and Ta5d6ff`Converter<PathEntry>` for ArduinoJson |

| Ta5d6ff`lib/microReticulum/src/Reticulum.h` | Updated Ta5d6ff`get_path_table()` return type |

| Ta5d6ff`lib/microReticulum/src/Reticulum.cpp` | Updated Ta5d6ff`get_path_table()` and Ta5d6ff`drop_all_via()` for deque iteration |

| Ta5d6ff`lib/microReticulum/src/Link.cpp` | Added missing Ta5d6ff`Link::attached_interface()` const getter |

| Ta5d6ff`RNode_Firmware.ino` | All interfaces → Ta5d6ff`MODE_FULL`; reduced path table caps; wired Ta5d6ff`clear_caches_in_memory()` into heap relief |